Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.29%. Comparing base (e923387) to head (0e83c56).
Report is 8 commits behind head on main.

✅ All tests successful. No failed tests found.

@@           Coverage Diff           @@
##             main      #26   +/-   ##
=======================================
  Coverage   82.29%   82.29%           
=======================================
  Files          18       18           
  Lines        1079     1079           
  Branches      174      174           
=======================================
  Hits          888      888           
  Misses        166      166           
  Partials       25       25           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

There's a bit of an impedance mismatch between the way the dax_obscore SIA code wants to handle collections and the collection structure of LSSTComCam/runs/DRP/... and LSSTComCam/prompt/...

The SIA query is using a find_first search, which isn't going to work with re-runs having the same data IDs in multiple collections in LSSTComCam/runs/DRP -- since it will be arbitrary which collection is first. My intuition would be that it should not be using find_first, so that we just find all images in all collections. But there wouldn't be any way to distinguish images from different collections from each other in the results, currently.

Perhaps better would be if the Butler collections were exposed as ObsCore collections, so that the caller could specify which collections they wanted to search in. But the code isn't set up to do that right now -- the ObsCore collection name is just a static string defined in the config file.

We might consider creating some chained collections in the embargo repo to serve as explicit top-level collections for SIA searches, which would make things easier to administer.

In any case the LSSTComCam/nightlyValidation chained collection seems well behaved with the current code and at least has raw, postISRCCD, and calexp in it. So that should be sufficient to unblock Stelios and Trey for the moment.

Tweaking the SIA code itself is a bigger scope than I want to jump in and help with right now, so getting access to the other collections might need to wait until Tim is back.

Oh, ok, if the collections here are all queried together in the end anyway, then I'd recommend using the most recent DRP collection as the only one until we can rework things as you've described, as that runs a superset of the tasks in nightly-validation with consistent code and configuration, and now that we're not on-sky anymore there's no real value to being able to automatically pick up new processing from prompt or nightly-validation.

OK, that should get us a little further. The existing SIA code has the ability to override the collection config on the fly, so Stelios can add some code to the SIA service to locate and use the latest DRP collection.

Or I guess we could ask the campaign team to maintain a chained collection with the latest DRP as the only child.

cc/ @stvoutsin

OK, that should get us a little further. The existing SIA code has the ability to override the collection config on the fly, so Stelios can add some code to the SIA service to locate and use the latest DRP collection.

Just to make sure I understand what this means:

• Locate the latest, something like:

drp_collections = butler.registry.queryCollections("LSSTComCam/runs/DRP/%")
latest_drp = max(drp_collections)

(Don't know if this actually works)

• Use the latest
  Replace what is in collections in the obscore_config with this before sending it off to siav2_query? (or append to it?)

Where is the DRP collection path/string configured? Do I parse this from the obscore_config you will be publishing here, or is it something that needs to be part of the application configuration?

It would be something approximately like that, yeah. It would have to be a hack that's part of the application configuration.

But hopefully you don't have to do anything -- after no-Slack-day ends I'm going to ask Campaign Management to maintain a collection for this, which is easier for you and will give us a way to keep failed and in-progress pipeline runs out of SIA.

With the code in this PR as-is, you should have enough to start working on getting the service set up at USDF.

since you're looking at this PR right now:

I think the configuration will be pretty much identical to IDF, except the Butler repository is called embargo, and the Butler server domain names are the USDF RSP domains instead of the IDF ones.

since you're looking at this PR right now:

I think the configuration will be pretty much identical to IDF, except the Butler repository is called embargo, and the Butler server domain names are the USDF RSP domains instead of the IDF ones.

Ok sounds good, a preliminary branch with the phalanx/sia configuration for this is here:
lsst-sqre/phalanx@main...tickets/DM-48143
Although I'm not sure if the butler path is correct there, I might add you as a reviewer if thats ok once I PR

Campaign Management has agreed to manage a collection LSSTComCam/SIA pointing to the latest DRP. So that is what we'll serve from SIA initially. We'll need to revisit this once we start taking LSSTCam data.

I am going to merge this so that it goes into tonight's build of dax_obscore. Gregory, if any of the IVOA bits are not right let me know and I'll fix them in a later PR.